out_kinesis_firehose: enable HTTP debugging for AWS client requests #11015

runderwo · 2025-10-13T17:48:23Z

Previously, there was no connection from the HTTP client created by the AWS library to the built-in HTTP debugging callbacks. This change creates that path, so that HTTP traffic created by AWS HTTP clients can be debugged in the usual way (by enabling HTTP debugging at build time and supplying config variables in the output block).

Without this change there is zero visibility into AWS HTTP sessions even when HTTP debugging is enabled, which violates the principle of least surprise to the user, and also as a result it's difficult to get help from AWS support because they tend to blame the client rather than the service until malformed requests are ruled out.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

To enable HTTP debugging output for an AWS service, add one or more of the following to an output section:

      _debug.http.request_headers: true
      _debug.http.request_payload: true
      _debug.http.response_headers: true
      _debug.http.response_payload: true

Debug log output from testing the change

Relevant log snippet (sanitized):

[2025/10/13 16:06:09] [debug] [output:kinesis_firehose:kinesis_firehose.XXX] firehose:PutRecordBatch: events=5, payload=2729 bytes
[2025/10/13 16:06:09] [debug] [output:kinesis_firehose:kinesis_firehose.XXX] Sending log records to delivery stream XXX
[2025/10/13 16:06:09] [debug] [upstream] KA connection #569 to us-east-1.aws.x.y.z:15173 is connected
[2025/10/13 16:06:09] [debug] [http_client] not using http_proxy for header
[2025/10/13 16:06:09] [debug] [http] request headers
POST / HTTP/1.1
Host: us-east-1.aws.x.y.z:1234
Content-Length: 2729
User-Agent: aws-fluent-bit-plugin
Content-Type: application/x-amz-json-1.1
X-Amz-Target: Firehose_20150804.PutRecordBatch
x-amz-date: 20251013T160609Z
Authorization: AWS4-HMAC-SHA256 Credential=XXX
Connection: keep-alive


[2025/10/13 16:06:09] [debug] [http] request payload (2729 bytes)
{"DeliveryStreamName":"atlas-kinesis-splunk","Records":[{"Data":"<snip>"},{"Data":"<snip>"},{"Data":"<snip>"},{"Data":"<snip>"}]}
[2025/10/13 16:06:09] [debug] [http] response headers
HTTP/1.1 200 OK
x-amzn-RequestId: xxx
x-amz-id-2: yyy
Content-Type: application/x-amz-json-1.1
Content-Length: 1259
Date: Mon, 13 Oct 2025 16:06:09 GMT


[2025/10/13 16:06:09] [debug] [http] response payload (1259 bytes)
{"Encrypted":false,"FailedPutCount":0,"RequestResponses":[{"RecordId":"<snip>"},{"RecordId":"<snip>"},{"RecordId":"<snip>"},{"RecordId":"<snip>"},{"RecordId":"<snip>"}]}

Attached Valgrind output that shows no leaks or memory corruption was found

==362092== 
==362092== HEAP SUMMARY:
==362092==     in use at exit: 13,874 bytes in 49 blocks
==362092==   total heap usage: 247,903 allocs, 247,854 frees, 58,923,676 bytes allocated
==362092== 
==362092== LEAK SUMMARY:
==362092==    definitely lost: 0 bytes in 0 blocks
==362092==    indirectly lost: 0 bytes in 0 blocks
==362092==      possibly lost: 0 bytes in 0 blocks
==362092==    still reachable: 13,874 bytes in 49 blocks
==362092==         suppressed: 0 bytes in 0 blocks
==362092== Reachable blocks (those to which a pointer was found) are not shown.
==362092== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==362092== 
==362092== For lists of detected and suppressed errors, rerun with: -s
==362092== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Documentation

Documentation required for this feature

I can't tell if the HTTP debug callback configuration keys are documented somewhere; I had to find them and guess how to use them by reading code.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

New Features
- Added a callback context to the AWS client to enable richer HTTP request handling and debug hooks.
Refactor
- Integrated the AWS client callback into the HTTP request flow and output plugin debug setup.
Bug Fixes
- Fail-fast behavior when callback setup during AWS client initialization fails.
Chores
- Ensure proper cleanup of the callback resource during client teardown.

coderabbitai · 2025-10-13T17:50:58Z

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Added a public http_cb_ctx pointer to struct flb_aws_client. Creation now allocates this callback and returns NULL on allocation failure. Requests propagate the callback to the HTTP client. The callback is destroyed on client teardown. Firehose plugin calls HTTP client debug setup during init.

Changes

Cohort / File(s)	Summary
Public AWS client struct update `include/fluent-bit/flb_aws_util.h`	Added field `struct flb_callback *http_cb_ctx;` to `struct flb_aws_client` (placed after `debug_only`).
AWS client lifecycle and request wiring `src/aws/flb_aws_util.c`	`flb_aws_client_create()` initializes `client->http_cb_ctx` via `flb_callback_create("aws client")` and returns NULL on failure; `flb_aws_client_destroy()` calls `flb_callback_destroy()` on `http_cb_ctx`; `request_do()` assigns `c->cb_ctx = aws_client->http_cb_ctx` before using the HTTP client.
Firehose plugin: HTTP debug setup `plugins/out_kinesis_firehose/firehose.c`	Added `#include` for `flb_http_client_debug.h`; during `cb_firehose_init` calls `flb_http_client_debug_setup(ctx->firehose_client->http_cb_ctx, &ins->properties)` and logs an error / aborts init on negative return.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant App as Caller
  participant AWS as flb_aws_client_create()
  participant CB as flb_callback_create()
  participant CL as flb_aws_client

  App->>AWS: create()
  AWS->>CB: flb_callback_create("aws client")
  alt Callback created
    CB-->>AWS: flb_callback*
    AWS-->>App: flb_aws_client* (with http_cb_ctx)
  else Creation fails
    CB-->>AWS: NULL
    AWS-->>App: NULL (early return)
  end

sequenceDiagram
  autonumber
  participant CL as flb_aws_client
  participant RQ as request_do()
  participant HC as HTTP Client

  CL->>RQ: make request
  RQ->>RQ: c->cb_ctx = CL.http_cb_ctx
  RQ->>HC: init/send with cb_ctx
  HC-->>RQ: response / error
  RQ-->>CL: result

sequenceDiagram
  autonumber
  participant App as Caller
  participant D as flb_aws_client_destroy()
  participant CL as flb_aws_client

  App->>D: destroy(CL)
  D->>CL: flb_callback_destroy(CL->http_cb_ctx)
  D-->>App: done

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I twitch my whiskers, find a hook,
A tiny callback joins the book.
It hops into requests with cheer,
If creation fails, we stop here.
I tidy up and bound away—hooray! 🐇✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and accurately describes the main change: enabling HTTP debugging for AWS client requests in the Kinesis Firehose output plugin.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2b16b24 and 7e4d2c2.

📒 Files selected for processing (2)

include/fluent-bit/flb_aws_util.h (1 hunks)
src/aws/flb_aws_util.c (3 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

src/aws/flb_aws_util.c (1)

include/fluent-bit/flb_mem.h (1)

flb_free (126-128)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (30)

GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
GitHub Check: pr-compile-centos-7
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
GitHub Check: pr-compile-without-cxx (3.31.6)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
GitHub Check: PR - fuzzing test

🔇 Additional comments (2)

include/fluent-bit/flb_aws_util.h (1)

105-107: LGTM!

The new callback field is appropriately placed in the struct with a clear comment describing its purpose.

src/aws/flb_aws_util.c (1)

395-396: LGTM!

The callback is correctly wired into the HTTP client context for debugging purposes.

src/aws/flb_aws_util.c

edsiper · 2025-10-13T20:02:35Z

why is this change needed ?
what's the end-user outcome ?
fix DCO (sign-off)

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

cosmo0920

I added a nitpick comment.
The logic of PR is reasonable so we only need to follow contributor guideline.

src/aws/flb_aws_util.c

runderwo · 2025-10-21T18:35:44Z

I added a nitpick comment. The logic of PR is reasonable so we only need to follow contributor guideline.

Resolved that since the client name was considered acceptable.

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

coderabbitai

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 909b105 and 4daa282.

📒 Files selected for processing (1)

src/aws/flb_aws_util.c (3 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

src/aws/flb_aws_util.c (1)

include/fluent-bit/flb_mem.h (1)

flb_free (126-128)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (30)

GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
GitHub Check: pr-compile-without-cxx (3.31.6)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
GitHub Check: pr-compile-centos-7
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
GitHub Check: PR - fuzzing test

🔇 Additional comments (2)

src/aws/flb_aws_util.c (2)

271-276: LGTM! Proper error handling for callback creation.

The callback creation with error handling is correct. On failure, the code properly calls flb_errno(), frees the allocated client, and returns NULL, preventing memory leaks.

396-397: LGTM! Correct propagation of callback context.

The callback context is properly propagated to the HTTP client, enabling HTTP debugging for AWS requests as intended by this PR.

src/aws/flb_aws_util.c

runderwo · 2025-10-21T18:55:40Z

@cosmo0920 I'm unable to re-request review through the UI for some reason, so please re-review.

edsiper · 2025-11-06T22:02:32Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

src/aws/flb_aws_util.c

plugins/out_kinesis_firehose/firehose.c

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

runderwo requested review from PettitWesley, celalettin1286, edsiper, fujimotos, koleini, niedbalski and patrick-stephens as code owners October 13, 2025 17:48

github-actions bot added the docs-required label Oct 13, 2025

runderwo temporarily deployed to pr October 13, 2025 17:48 — with GitHub Actions Inactive

runderwo temporarily deployed to pr October 13, 2025 17:52 — with GitHub Actions Inactive

coderabbitai bot reviewed Oct 13, 2025

View reviewed changes

src/aws/flb_aws_util.c Outdated Show resolved Hide resolved

src/aws/flb_aws_util.c Outdated Show resolved Hide resolved

src/aws/flb_aws_util.c Outdated Show resolved Hide resolved

runderwo temporarily deployed to pr October 13, 2025 18:12 — with GitHub Actions Inactive

Enable HTTP debugging for AWS client requests

3863eea

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

runderwo force-pushed the enable-aws-http-debug branch from 7e4d2c2 to 3863eea Compare October 14, 2025 09:13

runderwo temporarily deployed to pr October 14, 2025 09:13 — with GitHub Actions Inactive

runderwo temporarily deployed to pr October 14, 2025 09:19 — with GitHub Actions Inactive

Apply code quality changes

7ac03a2

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

cosmo0920 requested changes Oct 20, 2025

View reviewed changes

src/aws/flb_aws_util.c Outdated Show resolved Hide resolved

runderwo had a problem deploying to pr October 21, 2025 18:35 — with GitHub Actions Failure

Remove fixme

4daa282

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

runderwo force-pushed the enable-aws-http-debug branch from 3246c32 to 4daa282 Compare October 21, 2025 18:36

runderwo had a problem deploying to pr October 21, 2025 18:36 — with GitHub Actions Failure

runderwo requested a review from cosmo0920 October 21, 2025 18:38

coderabbitai bot reviewed Oct 21, 2025

View reviewed changes

src/aws/flb_aws_util.c Outdated Show resolved Hide resolved

chatgpt-codex-connector bot reviewed Nov 6, 2025

View reviewed changes

src/aws/flb_aws_util.c Show resolved Hide resolved

cosmo0920 requested changes Nov 7, 2025

View reviewed changes

plugins/out_kinesis_firehose/firehose.c Show resolved Hide resolved

Merge branch 'fluent:master' into enable-aws-http-debug

c782199

runderwo requested a review from a team as a code owner November 7, 2025 17:01

runderwo had a problem deploying to pr November 7, 2025 17:01 — with GitHub Actions Failure

Address feedback and make debug plumbing more canonical

a31ed4e

Signed-off-by: Ryan Underwood <ryan.underwood@mongodb.com>

runderwo temporarily deployed to pr November 7, 2025 17:27 — with GitHub Actions Inactive

runderwo changed the title ~~Enable HTTP debugging for AWS client requests~~ out_kinesis_firehose: enable HTTP debugging for AWS client requests Nov 7, 2025

runderwo temporarily deployed to pr November 7, 2025 17:46 — with GitHub Actions Inactive

runderwo deployed to pr November 7, 2025 17:46 — with GitHub Actions Active

runderwo temporarily deployed to pr November 7, 2025 17:46 — with GitHub Actions Inactive

out_kinesis_firehose: enable HTTP debugging for AWS client requests #11015

Are you sure you want to change the base?

out_kinesis_firehose: enable HTTP debugging for AWS client requests #11015

Uh oh!

Conversation

runderwo commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Other AI code review bot(s) detected

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Poem

Pre-merge checks and finishing touches

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

edsiper commented Oct 13, 2025

Uh oh!

cosmo0920 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

runderwo commented Oct 21, 2025

Uh oh!

coderabbitai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

runderwo commented Oct 21, 2025

Uh oh!

edsiper commented Nov 6, 2025

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

runderwo commented Oct 13, 2025 •

edited

Loading

coderabbitai bot commented Oct 13, 2025 •

edited

Loading